Automating root cause analysis at scale: Multi-signal correlation for cloud native incident responseSantosh Balaranganathan and colleagues at Atlassian describe their automated root cause analysis system that treats incident diagnosis as a correlation problem across three dimensions: signal type (metrics, logs, traces), time, and service topology. The pipeline scopes the search using OpenTelemetry-derived dependency graphs, detects anomalies independently per signal, temporally aligns co-occurring anomalies into bundles, traverses the graph to determine causal direction, and emits ranked hypotheses with human-readable narratives so responders can validate and act quickly.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
InfoQ writes:
>"Atlassian has outlined a new approach to automating root cause analysis for large-scale cloud-native incidents, using correlation across metrics, logs, distributed traces, and service topology to generate ranked hypotheses about where failures originate and how they propagate"
>"Atlassian has outlined a new approach to automating root cause analysis for large-scale cloud-native incidents, using correlation across metrics, logs, distributed traces, and service topology to generate ranked hypotheses about where failures originate and how they propagate"
Anirudh Ramanathan writes that while Anthropic suggests code is no longer the primary bottleneck in development, organizations cannot adopt a single, rigid software development life cycle (SDLC) for all changes. Instead, effective management requires a variety of processes tailored to the risk and complexity of each change—ranging from simple documentation fixes to high-stakes schema migrations—utilizing state machines that react to external evidence rather than fixed workflows.
- A spec-driven approach uses written artifacts like intent documents and plans as versioned drivers for development.
- High-velocity code generation necessitates verification mechanisms (like hooks or automated tests) that provide deterministic gates.
- Effective AI governance requires evidence from outside the agent, such as test results from independent systems, to ensure quality at scale.
- A spec-driven approach uses written artifacts like intent documents and plans as versioned drivers for development.
- High-velocity code generation necessitates verification mechanisms (like hooks or automated tests) that provide deterministic gates.
- Effective AI governance requires evidence from outside the agent, such as test results from independent systems, to ensure quality at scale.
Data Annotation is seeking DevOps Engineers to evaluate how AI models handle CI/CD, infrastructure-as-code, containers, and cloud operations. The role involves identifying failures in model automation—such as insecure defaults or broken state files—and writing correct solutions that a platform engineer would trust.
- Compensation ranges from $75 to over $150 per hour based on difficulty.
- No degree is required for the position; hands-on experience with tools like Terraform, Kubernetes, or Docker is prioritized.
- The role offers flexible hours as an independent contractor working remotely.
- Compensation ranges from $75 to over $150 per hour based on difficulty.
- No degree is required for the position; hands-on experience with tools like Terraform, Kubernetes, or Docker is prioritized.
- The role offers flexible hours as an independent contractor working remotely.
Harness provides an Autonomous Software Development Life Cycle (SDLC) platform designed to accelerate software delivery through AI-driven automation, security testing, and cost management. The platform utilizes various agents—including Software Delivery, Security Testing, Runtime Protection, and Cost Management—to automate pipelines from commit to production while maintaining governance and visibility across the entire development lifecycle.
- Offers three levels of risk-based autonomy ranging from manual oversight to fully autonomous agent execution within set policies.
- Includes a Knowledge Graph that learns patterns from deployments, incidents, and cost events to improve recommendations.
- Provides specialized AI posture management to identify and protect against threats like prompt injection and text evasion.
- Enables automated cloud spend attribution by tracing tokens to specific teams or business outcomes.
- Offers three levels of risk-based autonomy ranging from manual oversight to fully autonomous agent execution within set policies.
- Includes a Knowledge Graph that learns patterns from deployments, incidents, and cost events to improve recommendations.
- Provides specialized AI posture management to identify and protect against threats like prompt injection and text evasion.
- Enables automated cloud spend attribution by tracing tokens to specific teams or business outcomes.
bex is an open-source, self-hostable PaaS that positions itself as an AI-native alternative to Render, letting developers push Git and receive a deployed URL on their own Kubernetes infrastructure. Coding agents operate as first-class users via MCP alongside the dashboard, CLI, REST, and GraphQL interfaces, all backed by a shared Go core. The platform uses a Kubernetes operator with Cluster API for machine provisioning, supports Render-style `render.yaml` Blueprints for declarative service definitions, and ships managed Postgres, Key Value, logs, metrics, autoscaling, custom domains with TLS, and SSH access.
- 471 stars, 50 forks, 9 contributors — including Claude, Cursor, and Copilot listed as named GitHub contributors
- Apache-2.0 licensed; explicitly marked "not ready for production workloads" (public alpha)
- Language split: Go 58.5%, TypeScript 32.9%, Shell 6.7%
- Internal "lego" Go workspace enforces a strict `operator → types ← backend` one-way dependency DAG
- Tracks Render compatibility via an evidence-backed "parity ledger" (ADR018) rather than marketing claims
- Local quickstart provisions a kind cluster + Cluster API with Docker-container machines as tenant nodes
- Includes an Expo mobile app for safe supervision workflows (App Store listing present)
- Commit history references agent-driven QA rounds (w4/w5/w6 workstreams) and live dashboard re-probes
- 471 stars, 50 forks, 9 contributors — including Claude, Cursor, and Copilot listed as named GitHub contributors
- Apache-2.0 licensed; explicitly marked "not ready for production workloads" (public alpha)
- Language split: Go 58.5%, TypeScript 32.9%, Shell 6.7%
- Internal "lego" Go workspace enforces a strict `operator → types ← backend` one-way dependency DAG
- Tracks Render compatibility via an evidence-backed "parity ledger" (ADR018) rather than marketing claims
- Local quickstart provisions a kind cluster + Cluster API with Docker-container machines as tenant nodes
- Includes an Expo mobile app for safe supervision workflows (App Store listing present)
- Commit history references agent-driven QA rounds (w4/w5/w6 workstreams) and live dashboard re-probes
Anurag Singh writes that providing Claude Code with read-only access to a SaaS application's server logs allowed the coding agent to identify and propose fixes for real performance issues. By observing error patterns, traces, and metrics directly within the environment rather than relying on manual bug reports, the agent was able to autonomously trace bugs back to specific lines of code across various files.
- The experiment highlights a shift toward AI agents joining the "on-call" workflow by inspecting live operational telemetry.
- To mitigate security risks, it is recommended using Model Context Protocol (MCP) servers to restrict an agent's tools to read-only actions.
- Major observability companies like Sentry and Datadog are already implementing similar features to automate root cause analysis and pull request generation.
- The experiment highlights a shift toward AI agents joining the "on-call" workflow by inspecting live operational telemetry.
- To mitigate security risks, it is recommended using Model Context Protocol (MCP) servers to restrict an agent's tools to read-only actions.
- Major observability companies like Sentry and Datadog are already implementing similar features to automate root cause analysis and pull request generation.
Over 160 production-ready skill documents (SKILL.md) for LLM coding agents like Claude Code, Cursor, and Codex.Each skill contains expert-level configs, CLI commands, troubleshooting guides, and ready-to-run scripts spanning six domains: DevOps (CI/CD, Kubernetes, observability), Security (vulnerability scanning, secrets, hardening), Infrastructure (AWS/Azure/GCP, networking, databases, local inference stacks), LLM engineering (agent evals, RAG infrastructure, inference scaling), Compliance (SOC2, HIPAA, GDPR, ISO 27001), and IT operations. Skills install in under a minute via the `npx skills add` CLI or a simple git clone, and the agent discovers, matches, and activates them on demand.
- Ships with notable niche skills absent from other repos: MCP server security, eBPF kernel-level observability, OpenTofu migration, agent-eval CI/CD gates, and multi-tenant LLM hosting with KEDA autoscaling.
- Ships with notable niche skills absent from other repos: MCP server security, eBPF kernel-level observability, OpenTofu migration, agent-eval CI/CD gates, and multi-tenant LLM hosting with KEDA autoscaling.
Anurag Singh replaced five Python scripts (backup, organizer, renamer, cleaner, watchdog) with a local LLM agent, which made errors the scripts didn't (wrong directories, skipped steps, false success reports).Each of the original scripts followed explicit rules through a scheduler; the agent instead added a longer inference chain (inspect, interpret, choose a tool, build a command, execute, review) to tasks that fixed logic already described completely, while also holding a loaded model in memory between runs.
- AutomationBench scores for frontier models remain well under 20%: GPT-5.6 Sol 18.1%, GPT-5.5 12.9%, Claude Opus 4.8 15.5%, Gemini 3.5 Flash 14.5%
- Granting an LLM system-level access creates a prompt-injection vector: a malicious file on disk could carry instructions the agent interprets as commands
- Singh's proposed fix: let the agent classify and route ambiguous requests, then hand off to a validator + fixed script for the actual filesystem action
- The five original scripts covered photo backup, extension-based Downloads sorting, file renaming, app-cache clearing, and a disk-threshold alert
- AutomationBench scores for frontier models remain well under 20%: GPT-5.6 Sol 18.1%, GPT-5.5 12.9%, Claude Opus 4.8 15.5%, Gemini 3.5 Flash 14.5%
- Granting an LLM system-level access creates a prompt-injection vector: a malicious file on disk could carry instructions the agent interprets as commands
- Singh's proposed fix: let the agent classify and route ambiguous requests, then hand off to a validator + fixed script for the actual filesystem action
- The five original scripts covered photo backup, extension-based Downloads sorting, file renaming, app-cache clearing, and a disk-threshold alert
Anurag Singh replaced his home lab cron scripts with Qwen3.5 9B using an agent harness with shell access. He expected contextual reasoning to be superior to rigid automation. The local model succeeded in identifying ballooned directories or judging if a container restart was needed, but it failed more often, sometimes stalling or silently skipping checks.He concluded that deterministic scripts remain the more dependable choice for routine tasks and pointed to n8n as a sensible middle ground when the friction is writing and maintaining code rather than the logic itself.
- A 9-billion-parameter local model needs several GB of RAM just to load weights, which is painful on a home server already running Docker, DNS, and other services.
- Singh's specific hardware ceiling: roughly 14B parameters on a 16 GB MacBook, maybe 32B on an M5 Pro, beyond which you need a dedicated rig.
- His suggested hybrid: let the local model read an error log and draft a short explanation, then have n8n relay that summary without granting the model permission to restart or modify anything.
The model's failure mode was not wrong commands but an inconsistent process—the same prompt and the same system state, yet different execution paths on successive runs.
- A 9-billion-parameter local model needs several GB of RAM just to load weights, which is painful on a home server already running Docker, DNS, and other services.
- Singh's specific hardware ceiling: roughly 14B parameters on a 16 GB MacBook, maybe 32B on an M5 Pro, beyond which you need a dedicated rig.
- His suggested hybrid: let the local model read an error log and draft a short explanation, then have n8n relay that summary without granting the model permission to restart or modify anything.
The model's failure mode was not wrong commands but an inconsistent process—the same prompt and the same system state, yet different execution paths on successive runs.
